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BACKGROUND 


Health information systems (HIS) represent a significant investment for global, International, and subnational 
groups and organizations that require evidence for accountability and informed decision making. However, 
despite such a compelling need for robust evidence of HIS function, these systems have not traditionally been 
a subject for rigorous study and evaluation. That lack limits learning, sharing, and cultivating best practices 
that can be shared more widely. 

MEASURE Evaluation, a five-year cooperative agreement with the United States Agency for International 
Development (USAID), helps countries improve HIS management, governance, and performance. In July 
2014, USAID asked MEASURE Evaluation to implement activities to justify and build an evidence base for 
which investments in HIS are effective and useful. In response, we developed an HIS Learning Agenda, 1 to 
explore what works to strengthen HIS. For the Learning Agenda, we seek to answer questions such as these: 
What are the factors and stages of progress in HIS performance and how are they measured? And what are 
the characteristics of a strong HIS? The project is also implementing activities to build the evidence base on 
HIS strengthening. We hope our efforts will contribute to evidence-informed global work to strengthen HIS 
and health outcomes. 


INTRODUCTION 

One of the many questions that arise in linking improvements to health information systems (HIS) to 
improved health outcomes is the aspect of coherent client care, or “continuity of care (COC).” COC is the 
process of providing good-quality healthcare, over time, to a client. Continuity refers to the degree to which 
care is linked among all providers and made coherent over time. COC is especially important in helping HIV- 
positive people stay in treatment, therefore contributing to the 90-90-90 goals described by the Joint United 
Nations Programme on HIV/AIDS (UNAIDS), but it is also important in other health arenas, such as 
maternal care and childhood immunizations. 

An essential element of proper HIS function is that it generates reliable information about population health 
Service needs and individual care needs, the latter so that clients receive COC for their health issues, no 
matter how many providers are involved. One activity in MEASURE Evaluation’s Learning Agenda is to 
synthesize what we know so far about how to strengthen HIS and the effects of those efforts. The focus of 
this synthesis report—which is one of a series—is to document what we are learning about how stronger HIS 
may improve COC for clients. Here we review what we know now about the effects of HIS strengthening on 
COC and what areas need research. 


METHODS 

We chose COC as a topic for exploration, because it can serve as a proxy for an improved health outcome 
resulting from HIS strengthening. The lack of a discipline to document how investments in HIS contribute to 
improved health outcomes means that the peer-reviewed literature offers scant evidence specific to our query. 


1 MEASURE Evaluation. (2015) Learning agenda. Chapel Hill, NC, USA: MEASURE Evaluation, University of North Carolina at 
Chapel Hill. Retrieved from https://www.measureevaluation.ora/resources/publications/fs-15-142 
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Therefore, to compile evidence for this synthesis, we have blended a review of published litcrature widi 
evidence from our own work to strengthen HIS. The results are discussed below. 

First, we conducted a literature review on the concept of COC to find out what evidence exists for linldng 
strong HIS with improved COC. We searched for peer-reviewed journal articles and gray literature using 
relevant databases (PubMed, Google scholar, OVID, and Global Health). We used search terms ("continuity 
of care," "continuity of patient care," "continuity," "care," and "care continuum") to define COC, the impact 
of HIS on COC, and the impact of COC on health outcomes. Most of the literature focuses on how 
interventions produce better health outcomes and does not describe the work applied to elements of the HIS 
that makes up the intervention (e.g., management, training, procedures, etc.). 

We also reviewed all MEASURE Evaluation activities, by examining quarterly reports and discussions with 
activity leads and other staff, to identify which ones addressed COC. Our aim was to explore how each of the 
HIS activities in each country was contributing to COC. We sought activities that met these criteria: activities 
being implemented in specific countries (as opposed to global activities), activities that had not experienced 
significant delays in implementation, and activities that provided examples of different ways that care is 
defined within the context of MEASURE Evaluation’s work. 

We chose to highlight four activities in three countries. 

1. Gender-based violence (GBV) referral system, in Botswana 

2. Referral strengthening, in Kenya 

3. Child-protection information system, in Kenya 

4. Family folder program, in Ethiopia 

Mapping to the HIS Strengthening Model 

Part of MEASURE Evaluation’s HIS Learning Agenda has been to develop a health information system 
strengthening model (HISSM) (Figure 1). The model illustrates the logical progression of the effects of HIS 
strengthening activities to improve management, data, and data use to improve health systems and health 
outcomes for people. One of the questions we are asking is what types of HIS investment—in which 
components of the model—are most effective for better outcomes. 
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Figure 1. HIS Strengthening Model (HISSM) 
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Souice: MEASURE Evaluohon. 2017. Chapel Hill, NC: MEASURE Evaluation, 
Carolina Populafion Center, Universityof North Carolina. 


This model is divided into several areas: the human element and contextual factors (bodi of wliich affect 
everything), die enabling environment (HIS leadership and governance and HIS management), information 
generation (data sources, data management, and information products and dissemination), and HIS 
performance (data quality and use). It is a starting point for framing how an HIS works, what we lmow now 
about contributions to improved outcomes, and where in the framework are the opportunities to learn more 
about strengthening the HIS. This model also helps us map MEASURE Evaluation’s activities to assess what 
part of HIS strengthening we are addressing. One of the premises of the model is that the HIS is a tool that 
serves the larger health system and it can only be as strong as the system it supports. 


By Strengthening Health Information Systems 9 


WHAT WE LEARNED FROM THE LITERATURE REVIEW 

How Is COC Defined? 


The concept of COC emerged in reaction to concerns about how client care models had evolved from a 
single provider (who takes care of a client over time) to team medical practices, plus a new emphasis on COC 
as a pillar of primary care (Starfield, Shi, & Macinko 2005). COC also became a concern regarding clients 
without medical homes who are seeing new providers for every disease or injury and for clients with 
comorbidities that ought to be managed by multiple providers (Haggerty, et al., 2003). 2 COC is important in 
the areas of prenatal, postnatal, and early childhood care. Attending first and fourth antenatal care 
appointments are indicators of continuity that are tracked in low- and middle-income countries. Early 
childhood immunizations often require multiple doses to be effective and are another health-seeking behavior 
that good COC can facilitate. 

In defining what constitutes good COC, the assumption was that the best care is provided by a single 
physician who manages all aspects of care and is familiar with the entire health history of a client. From the 
client side, COC means constancy of a single clinician to manage care, having a usual source or place of care, 
and keeping follow-up appointments. From the clinician’s point of view, good COC means the availability of 
client information in a medical record, providing seamless and coordinated care during transitions among 
providers, and having an ongoing commitment to the client (van Walraven, Oake, Jennings, & Forster, 2010). 

Three categories of COC have emerged from this thinking: provider continuity, information continuity, and 
management continuity (Flaggerty, et al., 2003). According to van Walraven, Oake, Jennings, and Forster (2010), 
provider continuity is defined as a relationship between a client and one provider over time. MEASURE 
Evaluation does not work at that level, so our focus was on the other two categories. Infonnation continuity 
means the facilitation of ongoing healthcare through the availability and use of data from past health 
encounters during treatment for current health-related events (Keenan & Aquilino, 1998; Sparbel & 
Anderson, 2000; Rusch, 1986). Management continuity (sometimes referred to as “care coordination”) means 
the coordinated delivery of care from different providers (Haggerty, et al., 2003; Keenan & Aquilino, 1998; 
Sparbel & Anderson, 2000). 

Other definitions blend two or all three categories. According to Cabana and Jee (2004), continuity refers to 
“care over time by a single individual or team of healthcare professionals and to effective and timely 
communication of health information.” Shortell (1976) defines continuity of medical care “as the extent to 
which Services are received as part of a coordinated and uninterrupted succession of events consistent with 
the medical care needs of clients.” 

How Does COC Improve Health Outcomes? 

While there is quite a bit of information on how COC may improve health outcomes, the direct evidence for 
this effect is more limited. A substantial portion of the evidence available uses outcome indicators that are 
intermediary outcomes (missed appointments, adherence to treatment, immunization, health Service use, etc.) 
and not actual health outcomes, such as the presence or absence of diabetes, hypertension, etc. (O’Malley, 
2004). COC is especially important to people who suffer from chronic diseases and require close follow-up 
(Breslau, 1982; Guthrie, Saultz, Freeman, & Haggerty, 2008; Laerum, 1992; Weyrauch, 1996). For these 


2 The main subspecialties that are concerned with promoting continuity of care are primary healthcare specialists and 
mental health professionals. This is largely because the illnesses presented in these subspecialties are typically associated 
with other comorbidities that make it imperative for coordination and continuity of care if clients are to receive effective 
holistic treatment. 
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people, all three types of COC are important in securing better health outcomes, because they all support 
efficiency and effectiveness. In South Korea, a study by Nam, et al., (2016) based on national health insurance 
claims, found that greater COC was associated with a lower risk of readmission of hypertensive clients. The 
authors suggested that the mechanism of impact was that COC improved treatment adherence, which 
facilitated the reduction in likelihood of readmissions. 

In a study on coordination of maternal and infant HIV Services conducted in Malawi, the authors found that 
the delivery of antenatal care (ANC), prevention of mother-to-child transmission programs, and early infant 
diagnosis in separate facilities is associated with delays in antiretroviral therapy (ART) treatment initiation, 
loss-to-follow-up, high mortality in infants with HIV, and increased transit costs. The authors proposed that 
Information continuity, through use of a unique client ID, would link clients to their medical records and ensure 
that—irrespective of where they seek care—their medical history would be complete across a continuum of 
Services and that fact would decrease the number of missed opportunities and save lives (Braun, et al., 2011). 
Information continuity helps prevent medical errors and safeguards client safety. It accomplishes these two 
goals by ensuring that up-to-date client health information is available at every clinical encounter and that the 
provider is aware of key details in a client’s history; previous medications; and possible adverse effects of 
newly prescribed medication, especially in emergencies (Burton, Anderson, & Kues, 2004). 

A study by Buescher, et al. (1991) in North Carolina evaluated the impact of care coordination (the 
management continuity category) on birth outcomes for pregnant women on Medicaid. The program assigned 
maternity care coordinators to eligible women to assist them in accessing Services that addressed medical 
needs plus nutritional, psychosocial, and financial resource needs. Using a retrospective cohort study design, 
health program data files were linked to certificates of live births to compare birth outcomes. Study results 
showed that negative birth outcomes (low birth weight and infant mortality rate) were more prevalent among 
women on Medicaid who did not receive maternity care coordination. 

Health Information Systems and Continuity of Care 

COC requires certain data sources: a system to collect and record client information (whether paper-based or 
electronic), a system to make this information available to providers within and outside of the originating 
practice, and a system to follow up with clients who have been referred to other providers and with those 
who have missed appointments. All this information is dependent on investments in an HIS. 

Individual medical records are a component of a strong HIS and are important, because they provide 
information summarizing a client’s history and previous care. Both electronic medical records (EMR) and 
electronic health records (EHR) can improve efficiencies, availability, and readability of health information 
and improve infomation continuity. EHRs can also improve management continuity, because the information 
can be shared more easily among providers outside the originating practice. Some electronic systems can also 
provide prompts to providers, which can minimize medical errors and encourage providers to be proactive in 
healthcare provision (e.g., a reminder to offer a flu vaccine to a client). Many of these types of investments 
have been shown to be associated with improved COC. 

For example, outside of the United States, providers are using client-held cards (sometimes referred to as 
logbooks, client travelling records, or shared care diaries) that contain the clients’ individual medical 
information. These are kept by the client, who presents the card to each provider visited. The provider then 
documents the Services provided and other medical information on the card (Ko, Turner, Jones, & Hill, 

2010). These cards have been used for family planning and ANC as well as chronic health conditions (Ko, 
Turner, Jones, & Hill, 2010; Schoevers, van den Muijsenbergh, & Lagro-Janssen, 2009; Turner & Fuller, 

2009). 
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A few studies have been published diat assess die impact of a powerful HIS tool—such as an HER—on 
COC in general or specifically on die three categories of COC. 3 The available studies use robust research 
designs—randomized control trial, quasi-experimental design, and a qualitative framework approach. The 
studies were conducted in developed and developing countries. Each study examined a different tool in a 
clinical setting but researchers were not focused on how the tool fit into the context of the larger HIS. 

Using the HIS strengthening model (HISSM) would have been useful in that regard. The HISSM highlights 
the main areas to be strengthened in the HIS: the enabling environment (HIS leadership and governance and 
HIS management) and information generation (data sources, data management, and information products 
and dissemination). The studies we reviewed focused solely on the tool, which is a type of data source. We 
are able to infer that an effective HIS improves COC, but the findings from the studies do not say that 
explicidy. 

In rural Kenya, Haskew, et al. (2015) assessed the impact of a cloud-based EMR system on the reduction of 
treatment gaps in the HIV treatment continuum. Using a pre- and post-intervention study design, the authors 
found that with the implementation of the EMR-based clinical decision support, the percentage of clients 
who were eligible for ART but were missed decreased significandy (30% pre-intervention versus 6% post- 
intervention). 

In the U.S., an evaluation of a computerized system to facilitate end-of-rotation handover and sign-out 
procedures for medical residents found that the system improved COC, by decreasing the number of clients 
missed during the following morning rounds because of oversights in handover (Van Eaton, Horvath, Lober, 
Rossini, & Pellegrini, 2005). A study that evaluated a care coordination-enhanced tele-home care 4 program for 
elderly veterans with chronic heart failure found improvements in weight, in shortness of breath rating, and in 
blood pressure (Schofield, et al., 2005). Another intervention assessed the impact of tele-monitoring on COC. 
This study was a qualitative assessment of clients with chronic obstructive pulmonary disease (COPD), their 
healthcare professionals, and the managers of those providers. Based on a thematic framework analysis of the 
data gathered from interviews, the study authors found that provider relationship continuity was associated 
with client satisfaction (Fairbrother, et al., 2012). One meta-analysis of client-held cards found evidence that 
these cards can help improve health outcomes and preventive behaviors but that the evidence is stronger for 
maternal and child health than for other health conditions (Turner & Fuller, 2011). 

WHAT WE HAVE LEARNED FROM MEASURE EVALUATION’S 
WORKON HIS AND COC 

We reviewed all of MEASURE Evaluation’s activities, including USAID associate awards, to determine how 
many address COC and HIS. We found several that address COC, even though it was not the explicit 
objective of the activity. MEASURE Evaluation’s work spans the entirety of the HISSM: developing core 
documents for data quality, national strategies, and training curriculum (enabling environment); developing 
ways to integrate data collection and referral tracking in DHIS 2 and developing and implementing electronic 
data collection tools (information generation); and assessing data quality and use for programs (HIS 
performance). 


3 Haskew, Re, Turner, Kimanga, Sirengo, & Sharif, 2015: Casey, et al., 2013: Fairbrother, Pinnock, Hanley, McCloughan, 
Sheikh, Pagliari, & McKinstry on behalf of the TELESCOT program team, 2012: Van Eaton, Horvath, Lober, Rossini, & 
Pellegrini, 2005: Schofield, et al., 2005. 

4 Tele-home care uses telecommunications—combining voice, video, and health data—to deliver healthcare Services to 
clients in their homes. 
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COC in MEASURE Evaluation’s work goes beyond the clinical setting and includes types of social care 
provided to individuals. For example, several of our activities focus on community-based information systems 
and community HIS diat incorporate a wide range of data (including HIV testing and treatment and data on 
other Services offered). Some activities pursue the dual purposes of improving HIS to improve case 
management and to collect information for monitoring and evaluation (M&E) reporting. MEASURE 
Evaluation is also worldng on global activities that focus on understanding how ART retention can be 
improved and how Software platforms such as DHIS 2 can be used for referrals. 

The examples that follow are organized to include the rationale for the activity and how each one maps to the 
components of the HISSM. 

Gender-Based Violence Referral System in Botswana 

OverView 

This activity focuses on COC for survivors of gender-based violence (GBV) and falls in the “information 
generation” area of the HISSM, because it developed a mobile phone-based data repository to facilitate 
referrals. This activity supports the government of Botswana through the Gender Affairs Department (GAD) 
in the Ministry of Nationality, Immigration, and Gender Affairs to implement a multisectoral GBV referral 
system. The objective of the activity is to ensure that GBV survivors have access to all the Services and care 
they need. A survivor may access Services from a health facility, the police, a nongovernmental organization 
(NGO) clinic, the department of social protection, the educational system, or the traditional justice system. 
Thus, the mobile referral system would connect providers who serve survivors to provide them with COC 
and to reduce the burden on the survivor to tell their story multiple times. This information system is not 
only a tool of the HIS but also a tool for other sectors that deal with GBV. 

The referral system was piloted August 1, 2016—July 31, 2017, in two urban and two rural areas. All Service 
providers were identified through a provider survey and were trained to use the system. Another 
implementing partner conducted community mobilization around GBV in the pilot areas. 

The Need that the MEASURE Evaluation Activity Meets in Botswana 

There is currently no national system to gather information on GBV. Fewer than half of all providers collect 
some information; and these providers use forms they created themselves. Therefore, care for GBV survivors 
is scattered and uncoordinated, and the system has created a heavy burden on survivors, on Service agencies, 
and on national programs. For example, Standard operating procedures (SOPs) for dealing with GBV in a 
multisectoral way beyond clinical care do not exist. Management Sciences for Health (MSH) is currendy 
developing an SOP for GBV in Botswana, and MEASURE Evaluation will finalize it in 2017. Findings on the 
informal referral system gathered from local and national stakeholder meetings showed that providers were 
losing clients who were traumatized from having to retell their stories multiple times. It also showed that 
client referral forms were not used consistendy or correcdy and that tracking clients or referrals was not easy, 
because it required Service providers to maintain a register and required motivated staff to do the follow-ups 
that are important in these cases. This situation meant that generating national statistics about GBV was a 
challenge and the country often relied on special studies to generate estimates. 

HISSM: Enabling Environment 

MEASURE Evaluation has worked with key stakeholders (the GAD, Ministry of Health [MOH], Department 
of Social Protection, Botswana Police Service, Department of Education, Tribal Administration, and relevant 
national and local NGOs and civil society organizations) and brokered partnerships with private partners, 
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such as mobile phone companies, to develop and implement die referral system. The GBV referral system 
will collect uniform information across providers. Mantra Solutions developed the mobile and web-based 
application and the data dashboards and databases for accessing and reporting data. MEASURE Evaluation 
developed and implemented the first four user trainings with the assistance of the GAD. To ensure 
sustainability, a training of trainers was conducted for the GAD and for two in-country subcontractors 
(Stepping Stones International and Women Against Rape). Training was then standardized so that consistent 
materials would be used for all future trainings on the system, including SOPs and GBV awareness 
assessments. 

HISSM: Information Generation 

The GBV referral system is a data source used to collect data at Service points, but it also generates 
information for stakeholders. The system produces regular reports on Standard, agreed-on indicators made 
available to stakeholders at the national and community levels, as well as to providers at site-level Service 
delivery points. When the electronic system is successfully implemented, it will reduce the burden of manual 
data entry for the workforce and will provide real-time data on referrals. Furthermore, it will provide a unique 
ID to each client that will facilitate tracking and monitoring Services received by each client per incident. An 
automated system also allows providers to follow up on cases they have referred. For example, once an 
incident is entered into the system, both the referring and receiving providers will get a short message Service 
(SMS) alert after 24 hours if the referral is incomplete. 

HISSM: Expected Effects on HIS Performance 

Data quality and data use are indicators of F1IS performance. The referral system is designed to produce high- 
quality data that can be used by Service providers and for national monitoring. Data entry has been designed 
to use pop-up menus and pre-populated fields for client and Service identification, which improves accuracy 
in data entry. Data use is expected to improve both at the provider level and at the national level. Providers 
can use the referral system to follow up on referrals they started and record information on Services provided. 
Dashboards were built to let the GAD see data as graphs and charts. 

This activity has an operations research component to assess its overall utility, its effectiveness at achieving 
stated objectives, and the viability of the eventual rollout of the system after the pilot stage. Ongoing 
monitoring uses both quantitative and qualitative data. Monthly user support meetings with Service providers 
are conducted to gather feedback and information in a focus group-like setting on technical, logistical, and 
training challenges and to facilitate discussion on other challenges in using the system and handling GBV 
cases within their institutions. Titis information is used to understand additional technical assistance and 
training needs and to facilitate updates or corrections to the pilot for better implementation. Site visits with 
Service providers occur regularly on a rotating basis to investigate any challenges, questions, or concerns and 
to provide ad hoc training or TA when needed. Additionally, any anomalies in referral system reporting (such 
as referrals that are not completed, or unusual trends in the data) trigger a provider site visit or follow-up to 
examine problems so they can be fixed. Monitoring also involves analyzing raw data from the referral system 
to discover general trends that may illuminate any additional areas of need or assistance. These data also help 
the GAD understand general challenges in GBV Service delivery and potential areas of concern for specific 
types of Services, all of which improve F1IS performance. The data also can be used for advocacy. 
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HISSM: Expected Effects on Health Outcomes 

The referral system is expected to improve the coordination of care for GBV survivors in the country and, 
dierefore, dieir health outcomes. The GBV referral system is a tool that spans the health system to offer 
protection to survivors but the tool will only be as strong as the system it supports. 

Because GBV reporting and Services encompass several agencies and stakeholders, the referral system will 
help providers identify what care any GBV survivor needs to access. It will reduce the burden on survivors by 
sparing them from repeating details of the incident to every provider, but it will also help each survivor seek 
all the necessary care. Raising awareness of GBV among both survivors and Service providers is, therefore, 
the first step toward a strong GBV referral system. 

Service providers must be able to assess the extent of GBV when they encounter survivors. Every Service 
provider, including the police, should have a GBV focal person who fully understands the scope of GBV and 
how to appropriately deal with a survivor. 

The improvement of the GBV referral system can be measured in outcomes for survivors but also can be 
measured through other indicators, such as knowledge among providers about referrals, about other 
providers in the network, and about where to refer; provider understanding of the use of the referral system 
application, dashboard, and other technical requirements; sufficient resources allocated to the GBV referral 
system from the health ministry; regular provider use of the referral system; regular completion of referrals; 
harmony between case management records and referral system information; and formalization of the system 
by relevant ministries. 

Challenges 

Potential barriers to the success of the GBV referral system are the lack of SOPs and lack of awareness and 
understanding of GBV by all providers. Even if SOPs are developed, providers still will need to be trained. 
Lack of stakeholder buy-in would be the most significant barrier to the successful implementation of the 
GBV referral system. Stakeholders at all levels have a role in the implementation and success of the system by 
ensuring the compliance of their Service providers, as well as cultivating an understanding of the utility of the 
system. Additionally, devoting resources of staff, time, and money will help ensure success. Lack of buy-in by 
the ministry responsible for GBV (in Botswana, the Ministry of Nationality, Immigration, and Gender 
Affairs) and insufficient dedication of human and financial resources for this activity has heretofore been a 
barrier to implementing the system. Technical barriers also have thwarted successful implementation. 
Although the actual operation of the mobile application has been shown to be easy and efficient, other 
mobile technical requirements have posed challenges, such as making sure the data allowance is not used up 
by nonapproved use of phones, that the phones are reloaded with data and SMS allowances every month, and 
that all phones are consistently charged and in working order. 

Referral Strengthening in Kenya 

OverView 

This activity centers on referral strengthening in Kenya through the associate award to MEASURE 
Evaluation—PIMA (MEval-PIMA). Referral strengthening is an example of an activity that spans the health 
system’s core functions of Service delivery and health information. This associate award illustrates the 
importance of working within the enabling environment of the HISSM. MEval-PIMA’s referral strengthening 
efforts have evolved since 2012, when the focus was on supporting health sector referral performance 
monitoring. This focus first shifted to providing more support for health-sector referral systems strengthening 
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and, subsequently, strengthening efforts were more specifically aimed at HIV referral systems and linkage to 
care for HlV-positive clients. MEval-PIMA has worked at the national level and in 10 target counties to 
strengthen structures for policy and coordination of referral systems, build the capacity of health workers to 
manage and monitor referral systems and provide linkage to care, and improve performance monitoring and 
data use for decision making. MEval-PIMA has engaged in this work alongside public, private, and faith- 
based providers. 

Referral system strengthening (RSS) is one part of MEval-PIMA’s larger mandate in Kenya to improve the 
availability of good-quality health information at national and subnational levels. 5 As the first step, MEval- 
PIMA assessed HIS systems at all levels to determine the strategy for improvement. For RSS, it was 
determined that a strong referral system should (1) be documented well, meaning it can be described the same 
way by different people; (2) run by itself, meaning that all players are aware of their responsibilities and 
comply with their roles; (3) have a feedback loop for learning and improvement; (4) be measurable or 
trackable; and (5) be diverse and allow for multiple approaches and innovations. MEval-PIMA has worked in 
each of these five domains with the government of Kenya and other development partners. 

The Need that the MEASURE Evaluation Activity Meets in Kenya 

The RSS initiative is a response to system weaknesses, such as a high proportion of clients receiving care at 
secondary and tertiary facilities when care at a primary health center would be appropriate and less expensive. 
A pilot implementation of a global toolkit for the Rapid Monitoring of AIDS Referral System 6 in two districts 
in Nyandarua County found a lack of quality data to measure the completion of referrals among clients at all 
levels of the health system, including the community level. The lack of data was attributed to a lack of referral 
performance monitoring systems, lack of data ownership and data use, and lack of a data management 
system. Additionally, an assessment undertaken in eight counties in 2013 showed that mechanisms to collect 
referral data were weak, that documentation of referrals was poor, tracking a referred client through the 
system was difficult, referral data were never analyzed or used systematically in decision making, referral 
coordination structures did not exist, and processes for referrals varied across facilities. MEval-PIMA has 
supported several initiatives to support referrals tracking, including defining priority indicators for referral 
performance monitoring, revising Kenya Ministry of Health (MOH) routine registers and DHIS 2 to include 
referral data elements and support for data collection on key indicators, and analyzing and using data for 
decision making. 

HISSM: Enabling Environment 

MEval-PIMA has worked to improve the enabling environment for strengthening referrals in Kenya. First, it 
aimed to improve the availability of policy and coordination structures to guide RSS. Through MEval-PIMA’s 
advocacy at the national level, a Referrals Unit was formed at the MOH to oversee referral strengthening and 
to monitor referrals nationally. MEval-PIMA supported the development of a national referral strategy, 
referral guidelines, formation of a referral systems technical working group (TWG), establishment of referral 
coordination structures at the subnational level, and customization of the referral strategy for county-level 
implementation. The National Referral Guidelines are the basis for clear documentation of referral standards 
in Kenya and are an important piece of the enabling environment component of the national HIS. The 
guidelines outline the structure and organization of the referral system; roles and responsibilities at each level, 
from the MOH to the referring and receiving facilities; management of referrals; coordination of referral 
Services; and monitoring of the referral system. In addition to the referral TWG, stakeholder forums were 

5 Other systems that are being supported are the community HIS and civil registration/vital statistics. 

6 https://www.measureevaluation.orq/our-work/hiv-aids/rapid-monitorina-of-aids-referrals-tested 
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conducted in selected counties to enhance coordination in die county referral system, provide a setting to 
discuss referral data and challenges in the referral process, and to review progress on the RSS action plans. 
Some counties established referral units, and MEval-PIMA provided technical support to establish the roles 
and responsibilities of these units and to build their capacity. 

The second step was to build the capacity of health workers to manage and monitor referral system 
performance. MEval-PIMA supported development of a training curriculum on referral systems, which has 
been used to train more than 2,000 health workers in 10 counties, supported by MEval-PIMA. The World 
Health Organization (WHO) and the United Nations International Children’s Fund (UNICEF) are using the 
curriculum to train staff in 11 more counties. MEval-PIMA also supported the development of an abridged 
curriculum on HIV referrals and linkage to care that has been used to train county-level mentors. These 
mentors guide health workers at the facility level on the management and performance monitoring of HIV 
referrals and linkage to care. 

HISSM: Information Generation 

This activity did not focus on creating or managing specific data sources, but it supported forums and 
mentorship that focused on the collection, analysis, and use of referral data, which have the effect of 
improving information products. The goal was to enhance collaboration among Service providers, promote 
ownership of activities to improve referrals in the target counties, and discuss data and challenges. The main 
objective of mentorship visits, which were conducted in collaboration with the county and subcounty 
management, was to enhance referral performance monitoring through technical support for facility 
personnel on how to improve collection, analysis, and utilization of quality referral data—all important 
components of an HIS. 

HISSM: Expected Effects on HIS Performance 

These activities would be expected to lead to increased data quality and use, which is how the HISSM defines 
HIS performance. Baseline data were collected in 2012, and follow-up data in 2014. Findings at key sites 
show that many issues remain regarding the quality of data entry in registers using the referral indicator 
columns, but monitoring has identified Solutions. As part of its work to strengthen the HIV referral network, 
MEval-PIMA worked with partners to select sentinel facilities in the target counties and conducted baseline 
assessments of those—involving data abstraction, questionnaires for staff, and focus group discussions—to 
determine the State of the referral system. Referral directories were created and validated in selected counties. 
Once the revised RSS was introduced, mentorship visits to high-volume health facilities were conducted to 
strengthen health workers’ capacity to initiate and manage referrals and monitor performance. Data on key 
referral indicators were collected during the mentorship visits, and data analyzed and displayed at the facility 
so performance on all indicators could be discussed and action plans for improvement developed. 

HISSM: Expecfed Effecfs on Health Outcomes 

A strong RSS is expected to have positive effects on providing HIV Services—and therefore improved health 
outcomes for HlV-positive clients. A strong referral system allows healthcare workers to correcdy link clients 
to needed care and allows facilities to track referrals to ensure follow-up. These improvements help ensure 
that clients receive Services in the appropriate Service delivery level. These outcomes are possible because all 
health workers have Standard forms, have guidelines to folio w, know the network of providers to refer to, 
and have been trained to analyze and use data to make decisions. Linking clients and keeping them in care 
contributes to achieving 90-90-90 goals, especially if all health programs adopt this model of a strong RSS for 
HIV Services. 
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Challenges 

The overall weakness of a health system would be the main barrier to successful implementation of RSS. The 
functioning of a referral system reflects the performance of the health system overall and affects the quality of 
care it delivers. While RSS is important, it is only one element of providing good-quality Services. Clinical staff 
who are trained and motivated, facilities that are clean and well maintained, access to laboratory Services, and 
availability of required medical commodities are necessary elements, as well. 

The Child Protection Information Management System in Kenya 

OverView 

The development of a child protection information-management system (CPIMS) in Kenya—an electronic 
system used to capture and share information on child protection data—is another example of MEASURE 
Evaluation’s work in COC that goes beyond the clinical setting. MEval-PIMA supported the Department of 
Children Services (DCS) to establish an electronic national child protection database that would be 
sustainable. This activity required work in the enabling environment, information generation, and HIS 
performance areas of the HISSM. The framework for Kenya’s national child protection system was designed 
to promote the welfare and well-being of children, by helping to prevent violence, abuse, neglect, and 
exploitation and addressing instances that occur. The child protection system is meant to ensure prompt and 
coordinated action to prevent further harm and to refer threatened children to the Services they need. The 
activity has the following components: systems and stakeholder assessments, system upgrades, user training, 
participation in and strengthening of a stakeholder coordination forum, and enhancement of the child 
protection data from CPIMS. 

The Need that the MEASURE Evaluation Activity Meets in Kenya 

This activity was proposed to help coordinate the collection and transfer of information about individual 
vulnerable children to all actors in child protection. Child protection falls under certain line ministries within 
the Kenyan government (health, interior, justice), each of which have their own systems to track child 
protection, none of them interoperable. A paper-based system produced low-quality data because of 
duplication of records and could not quickly aggregate data for sharing among stakeholders. 

HISSM: Enabling Environment 

Two national child protection systems had been tried before in Kenya and had failed. Reviews of these two 
systems concluded that they failed because of the following factors: (1) the inadequate involvement of local 
stakeholders in the design and implementation of the database to ensure sustainability past the duration of the 
initial project; (2) minimal levels of acceptance among users; (3) limited human resource capacity, especially in 
information and Communications technology (ICT) to manage the system; and (4) a rollout strategy that was 
not comprehensive. This information was used to help develop the new CPIMS, with one of the first steps 
being to develop a system design roadmap and sustainability plan. Titis was followed by an ICT audit. Once 
the first version of the system was developed, a user acceptance test was conducted to inform system 
changes. 

Given the challenges encountered by the two previous attempts, it was decided that a key to CPIMS success 
was wide stakeholder engagement to identify information needs to inform system development. The DCS did 
not have an M&E unit or M&E plan for child protection, which fact rnade it a challenge to determine what 
information should be collected. Therefore, an M&E unit was created to help oversee CPIMS, and MEval- 
PIMA helped develop an M&E plan using CPIMS as a data source. It is expected that DCS should own the 
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process, taking the lead role in managing the system while working with other partners for continuous 
support. MEval-PIMA also supported die development of system security and data quality guidelines. 

Several governance structures were established to inform the process of setting up the CPIMS: the ICT 
taskforce meets monthly, made up of the information technology staff from the stakeholders who manage 
the day-to-day development and implementation of the system; the CPIMS TWG meets every two months; 
and a steering committee chaired by the DCS meets every six months. 

Creating a training curriculum and training implementation are important components of a successful rollout 
of any information system within an HIS. For the CPIMS, user trainings take five days. The first three of 10 
counties were trained by the government of Kenya and MEval-PIMA and CPIMS launched in those counties 
in August 2016. Seven more counties were launched by the end of October 2016. 

HISSM: Information Generation 

The CPIMS is a data source with its own management structure to disseminate information. It includes 
modules for registries (organization and personnel), forms, and reports. As envisioned, the CPIMS will be a 
gateway for all who engage in child protection Services in Kenya to share relevant information, reducing 
duplication and delays in data sharing. It will also enable the DCS and the National Coalition of Children’s 
Rights and Child Protection to handle some of their data collection and M&E needs for child programs. 

HISSM: Expected Effects on HIS Performance 

A functional CPIMS is expected to improve data quality and use—and, therefore, HIS performance— 
because it can collect and aggregate data from all relevant actors in child protection in a timely manner. This 
means the data will be more complete and available earlier for decision making at all levels of child 
protection—from the field, when dealing with an individual case, to the national level. This expectation relies 
on the CPIMS being properly used. MEval-PIMA put several mechanisms in place to ensure this. Monthly 
site visits are conducted to assess how CPIMS use is progressing and to address any challenges. CPIMS 
allows supervisors to see what sites have not logged in so they can address tliis in site visits. Technical 
support is provided by the ICT authority and the DCS. A community of practice at the county level can help 
sites to resolve issues among themselves. Each county established a “WhatsApp” group for posting questions 
or issues they encounter. CPIMS champions have been identified to take the lead in resolving questions, and, 
during the pilot, it was found that the group can usually resolve the issues encountered or can seek help from 
the technical team (one person from the government and one from MEval-PIMA) with any unresolved 
questions. Data are monitored as they come in and data audits and reviews monitor data quality. The 
redundancy of parallel electronic and paper systems will aid in monitoring data quality. This, however, 
bespeaks hesitation to rely solely on an electronic system. 

HISSM: Expected Effects on Health Outcomes 

A strong CPIMS is expected to improve case management and Services provided to individual children, 
thereby improving their health outcomes. It also is expected to improve the capacity of the national program 
to monitor and report on achievements and gaps in child protection, thereby contributing to the robustness 
of the HIS. A strong CPIMS should have the following attributes: the ability to monitor all Services provided 
to a child by any provider (educational, psychosocial support, health, shelter, legal and protection, and food 
and nutrition); capacity for data visualization; flexibility in its web-based platform for batch uploading of data 
to optimize Internet connectivity; the capability of collecting longitudinal records; full interoperability with 
other systems in child protection; and the ability to link with other phone and mobile applications. 
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Challenges 

Some challenges and barriers to effectively implementing CPIMS are the potential to rush to enact an 
electronic solution before having an adequate enabling environment, sudi as a child protection M&E plan 
diat describes die information diat needs to be collected. Setting realistic timelines and adjusting the 
expectations of stakeholders has been a challenge. Stakeholder involvement is critical, and it is important that 
all staff involved with the CPIMS see its usefulness. The field staff must be transformed into users of the 
system and not just producers of data, so promoting data use will be an important challenge. Access to power 
and Internet remain a challenge at the county level, as does staff capacity. Continuous reinforcement of 
training is necessary to ensure people use the system correcdy. 

Family Folders in Ethiopia 

OverView 

This activity was accomplished during 2008—2014 of the MEASURE Evaluation project and is an example of 
the use of a paper-based system to improve COC in a resource-poor setting. It is another activity that worked 
in several areas of the HISSM: enabling environment, information generation, and HIS performance. 
MEASURE Evaluation supported Ethiopia to transform its national health management information system 
(HMIS), which receives data from a robust community health information system (CHIS). The centerpiece of 
this CHIS is the “family folder.” The family folder, which covers every household in the community, is a 
client flling system used at rural health posts that provides data for the CHIS. Two health extension workers 
(HEWs) per health post serve roughly 5,000 people. In what is called the “enumeration” phase, HEWs assess 
the households in their catchment area to collect information such as the age and sex of household members, 
health behaviors, and the availability of water and sanitation. Once the initial family folder is completed, the 
HEW updates the health card when family members come to the health post for Services or after the HEW 
visits the home. 

The Need that the MEASURE Evaluation Activity Meets in Ethiopia 

Prior to the family folder, no Standard information system existed for HEWs to use in their work with their 
communities. HEWs relied on registers provided by local health offices or NGOs and were expected to carry 
notebooks to record Services when they visited households and then transfer their notes to the registers in the 
health post. The entire national HMIS suffered from a lack of Standard definitions for data elements and 
basic indicators. 

HISSM: Enabling Environment 

Commitment from the government of Ethiopia was critical in the uniform implementation of the family 
folder system rollout throughout the country. Uniform implementation also involved effective coordination 
and collaboration among health sector actors to support the CHIS. Partnering with local networks was 
especially critical during the enumeration phase. The Kebele (village-level administrative unit) and the Health 
Development Army (a term used in Ethiopia for a cadre of health workers) were actively involved in this 
process, which was a massive undertaking. The community valued the effort, which helped in the motivation 
and mobilization to accomplish the task. 

MEASURE Evaluation assisted the Federal Ministry of Health, Ethiopia (FMOH) to roli out the CHIS 
family folder system nationwide. The main components of this activity were (a) developing and field-testing 
the operational guidelines for the use of family folders at the health posts, (b) developing a national training 
manual on family folders for HEWs, (c) training providers on a family-folder-centered CHIS, (d) establishing 
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a supportive environment, by building the mentoring capacity of the HEW supervisors, and (e) tclling die 
success stories of die family folders at national and international forums. One of die innovations diat 
MEASURE Evaluation introduced was die tickler file system, which organizes client records according to die 
future date diat a follow-up visit will be required. After each client visit, the HEW records a date for the next 
expected visit for the specific household member. The health card is removed from the family folder at the 
health post and placed in a box corresponding to the month of the next appointment. At the beginning of 
each month, the HEW reviews the cards in the box and prepares appointment reminders or conducts 
household visits. 

The family folder system was implemented in stages. The design phase involved the FMOH working with 
implementing partners to determine what information the family folder would collect and what the folder 
would look like. This design was reviewed by technical experts. However, because not all implementers were 
involved in the design, there was confusion about how to implement the system and so operational guidelines 
were devised to ensure uniform implementation at the community level. The FMOH formed a TWG that 
came up with options for using the system. These options were pilot-tested, the most feasible option was 
selected, and operational guidelines and training manuals were developed. This pilot was important, because it 
gathered feedback froni HEWs, who would be the users of this system. National scale-up was the last step. 

HISSM: Information Generation 

The family folder has five parts: (1) Identification, (2) household description, (3) household characteristics, (4) 
documentation of health-education trainings of household members conducted by HEWs and household 
status, and (5) documentation of a household’s implementation of the lessons froni the trainings. The outside 
of the folder contains information on drinking water sources, number of insecticide-treated bed nets, and 
latrine characteristics. Inside the family folder, a health card for each household member has information on 
health Services, including immunizations, family planning, and tuberculosis treatment. The family folder 
offers the HEW a one-stop source for review of all the Services received by everyone in the family. 

HISSM: Expected Effects on HIS Performance 

A successful family folder should compile data that is easy for HEWs to use and understand. It should 
provide them with information as they provide Services to the community. The family folder can also be used 
by supervisors for data quality checks and assessment of the quality of care provided, which improves the 
HIS and promotes objective mentoring of the HEWs. Data generated by family folders is transferred to the 
national HMIS for analysis of how the health system is performing at the community level. 

The family folder system has improved the quality of data produced and facilitated data use at the client and 
community levels and beyond. The Standard forms help HEWs collect data uniformly and completely for 
each household and the tickler file system helps them identify people who need follow-up Services, especially 
for pregnancy care, immunization, and family planning. This information reminds HEWs of follow-up visits 
to clients and helps them identify Service defaulters—which contributes to improvements in COC for these 
people. The additional information about household characteristics also helps HEWs tailor health training 
topics to the needs of a specific household. 

Health post staff conduct a regular data quality self-assessment using a lot quality assurance sampling survey 
(LQAS), which is a rapid and inexpensive approach to data collection primarily used for M&E of a specific 
area or facility of interest. The government also has conducted data quality assessments using MEASURE 
Evaluation’s Routine Data Quality Assessment Tool. 
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HISSM: Expected Effects on Health Outcomes 

This activity has had many successes and was selected by USAID as one of the top 10 case studies of health 
system strengthening activities. 7 No formal evaluation has been done on the effects of the family folder 
system on health outcomes, but there is anecdotal evidence that the effects are beneficial. In just one example 
of how the family folder helped improve health outcomes in Ethiopia, HEWs in one locale realized that 
mothers were not bringing their children in for follow-up immunizations. This realization led to a house-to- 
house canvass to encourage mothers to complete immunizations. Because of this effort, more children were 
immunized in one month alone than had been immunized in the three months before. 

Challenges 

Ensuring that a continuous supply of CHIS materials is available in a timely manner at health posts is a 
challenge. Often, shortfalls in supply result from distribution challenges or a lack of understanding of how 
many materials to distribute. There is still some confusion about who is responsible for the production, 
distribution, and costs of CHIS materials for the family folders at health posts. Also, the continued use of 
parallel information systems within the FMOH and systems promoted by implementing partners creates 
more work for HEWs and is a threat to the continued viability of the CHIS. In some places, the CHIS is used 
only for specific Services (such as family planning, immunizations, or ANC). At the initial stage, there was 
variation in how well the HEWs understood the CHIS and how it works—both in terms of the quality of 
understanding by any one person and the consistency of understanding among people across the levels of the 
health system. Targeted training for HEWs and their supervisors (who also were found to misunderstand the 
CHIS), along with strong supportive supervision, helped minimize these effects. High turnover among 
HEWs and supervisors means training is a continuous process. One significant change for the CHIS currently 
is a shift from a paper-based to an electronic system. 


7 https://hssalobalcall.hsaccess.org/top-cases 
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DISCUSSION 

Conclusions from the Literature Review 


In this synthesis, we have compiled evidence from die literature suggesting that improved COC positively 
affects health outcomes. The documents and activities reviewed provide plausible evidence that investments 
in HIS improve COC and that the tools discussed here (electronic health records, patient-held cards, etc.) also 
strengthen the HIS. However, we cannot find direct evidence in the peer-reviewed literature about how the 
COC interventions are (or are not) strengthening components of the HIS. Still missing is documentation of 
effective packages of HIS strengthening that will improve COC. For example, the GBV referral system is a 
data source that facilitates better outcomes for survivors and generates information for stakeholders and 
decision makers—a result that can be said to strengthen HIS. But the connection is not explicidy made in the 
analysis described in the literature. 

Conclusions from MEASURE Evaluation’s Experience 

Our review of MEASURE Evaluation’s activities have shown effects on multiple components of HIS 
strengthening, as mapped to the HISSM, in areas such as leadership, governance, management, and 
information generation. They also have been shown to improve COC. For example, RSS improvements in 
Kenya were supported with mentorship visits, the main objectives of which were to enhance referral 
performance and monitoring through technical support. This support, in turn, improved referrals and 
strengthened the HIS through better data collection, analysis, and use of good-quality referral data—all 
important components of an HIS. 

Evidence from MEASURE Evaluation’s work in the four activities discussed have several themes in common 
that highlight what we are learning as these activities are implemented. The activities concern the 
implementation of two electronic information systems, a paper-based system, and an activity that focused on 
referral strengthening in general. Activity reports indicate that specific program-data needs have been 
identified and mechanisms to improve data quality and use have been implemented. However, questions 
remain about whether these interventions also improve HIS functioning and health systems overall. We still 
can’t be certain if the interventions result in better care and health outcomes, because—even though the 
activities were designed to achieve titis objective—the necessary empirical studies have not been conducted. 

Moreover, all the activities are operating in dynamic environments with rnany stakeholders and engage 
multiple areas of the HISSM. Even when the focus of an activity was to create a data source, many other 
components of the enabling environment may not have been specifically addressed, such as governance, 
finance, and clear identification of information needs. Therefore, the creation of a data source, for example, 
cannot be isolated from other important components of an enabling environment. And these components 
will influence the activity—either negatively or positively—whether or not they are taken into account. The 
inability to isolate parts when implementing an activity makes the work in health systems complex and 
underscores the challenge of working in a system in which one activity does not exist in a vacuum and is 
impacted by other core functions. 

Another theme the highlighted activities have in common is that they are all designed to meet two primary 
needs of stakeholders: (a) a system that is easy to use and that will contribute to COC in the provision of 
health Services and case management, and (b) a system that collects information that can be aggregated to 
meet M&E requirements and reporting for decision making. These needs are a difficult balance of Service 
provision and data collection that is reliable for decision making. An HIS must not interfere with the primary 



need to provide care. Managing an HIS requires careful consideration of how much information providers are 
asked to collect, how easy or difficult the system is to use, how much training is required, and what kind of 
support can be provided. Shortcomings in these areas can lead to the underuse, incorrect use, or complete 
user abandonment of the HIS system. The possibility of abandonment was highlighted in the CPIMS 
example, which was the third attempt to develop a system for child protection in Kenya. 

These examples also underscore the importance of vigorous stakeholder engagement for the success of an 
activity. Effective stakeholder engagement requires not only assessing different information needs of each 
stakeholder but also gathering input from the system’s future users during its design. Pilot-testing new 
systems can reveal inefficiencies and areas for improvement. It can also help determine how people will be 
trained on the system once it is revised. System users must be comfortable with their roles and have a clear 
understanding of the purpose. Users must have supportive supervision when new systems are implemented, 
to ensure they use the system correcdy and to answer their questions or concerns. For COC to be achieved, 
M&E goals to be met, Services to be provided, and data quality assured, frontline users must be comfortable 
with the system. 

This focus on the people associated with the system is part of the common theme of the human element in 
the HISSM, which includes not only the people involved in designing and using information systems but also 
the clients, or consumers, themselves. As already discussed, users of the system play an important role in 
success but, in many cases, consumers in need of COC also should have their needs considered in design. No 
system will work if clients are not aware of it or don’t follow up with the Services provided. For example, in 
Botswana, community mobilization around GBV is essential to the success of the referral system, because 
GBV survivors and people close to them need to be aware that (a) GBV is a problem, and (b) there is a 
system in place to provide assistance. Anything that can be done to improve consumer engagement will 
improve COC. 

Finally, a limitation of this synthesis is that three of MEASURE Evaluation’s four activities are in the pilot 
phase or are ongoing, so we are unable to assess success or any long-term effect on providing COC. It would 
be advisable to return to these activities to update what has been learned about the HIS and improving COC. 
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